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FIELD OF THE INVENTION 
35 The present invention relates to digital image processing, and more particularly to 

adjusting images produced by partially defective digital image sensors. 
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BACKGROUND OF THE INVENTION 



Digital imaging devices, such as a charge-coupled-device (CCD) or a 
Complementary Metal Oxide Semiconductor (CMOS) device, generate digital signals 
representative of an image they are exposed to. These devices contain an array of light 
detecting photosites which accumulate charge depending on the light energy projected 
onto them. The charges are measured in signal processing circuits to produce a digital 
data signal for each image pixel of a display device. 

During manufacturing or use some photosites are flawed and thereby produce a 
defective image signal. There exist a number of post production methods for determining 
flaws and making adjustments to the images produced by the image sensor. These 
methods are preformed by software on an image processing computer that receives the 
images from the image sensor. They focus on finding flaws in the produced image. 
Therefore, when the same image sensor is coupled to a different image processing 
computer, the image cannot be fixed unless that different image processing computer 
includes the same software. Providing this software on every image processing computer 
is costly and impractical. 

Therefore, there exists a need to produce an image sensor device with 
predetermined information regarding the status of photosites in order to make processing 
images generated by the sensor device more efficient. 



The present invention provides a method and computer program product for 
profiling a digital image sensor at production time and editing images generated by the 
digital image sensor according to the profile. The method includes exposing a digital 
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image sensor with an array of photosites to a test card, comparing an image signal 
generated by one or more of the photosites in the array, based on the exposure to the test 
card, to an expected image signal result for the one or more of the photosites for the test 
card, and generating a profile of the digital image sensor based on the comparison. The 
5 method is repeated for a plurality of different test cards. After the digital image sensor 
records an image, a processor internal or external to the digital image sensor adjusts the 
recorded image according to the stored profile and a compensation algorithm. 

As will be readily appreciated from the foregoing summary, the invention 
provides a system, method, and computer program product for efficiently and effectively 
10 determining the status of a digital image sensor (camera) and producing quality images 
according to the determined status. 



1 5 become more readily appreciated as the same becomes better understood by reference to 
the following detailed description, when taken in conjunction with the accompanying 
drawings, wherein: 



BRIEF DESCRIPTION OF THE DRAWINGS 



The foregoing aspects and many of the attendant advantages of this invention will 



FIGURES 1 and 2 are block diagrams of components of the present invention; 



FIGURES 3-5 are flow diagrams of a method for executing the present invention; 



20 



and 



FIGURES 6A-C and 7 are an example of the present invention. 
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DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENT 
As shown in FIGURE 1, a first embodiment of the present invention is a system 
10 that includes a digital image sensor 12 with internal memory 14. The digital image 
sensor 12 is coupled to an image processor 20 with internal processor memory 22. The 
5 digital image sensor 12, for example a CCD or CMOS digital video camera, is coupled to 
the image processor 20 when still or video images recorded by the sensor 12 are to be 
downloaded from the sensor 12 and processed. Stored within the digital image sensor 
memory 14 is a profile of sensor 12. The profile identifies the operable and inoperable 
photosites of the digital image sensor 12. The process for determining the sensor's 
10 profile is described below with reference to FIGURE 3. 

Stored within the image processor memory 22 is a compensation algorithm. An 
example compensation algorithm is described in more detail below with reference to 
FIGURE 5. In an alternate embodiment, the compensation algorithm may be stored 
within the digital image sensor memory 14, if it is necessary to convert the digital signal 
15 generated by the sensor 12 into an analog equivalent for transmission to a display device 
or other analog device, such as a VCR. The image processor 20 is preferably a user's 
personal computer that receives the image as recorded by the digital image sensor 12 and 
processes the recorded image according to the compensation algorithm. The 
compensation algorithm uses the sensor's profile and is independent of the sensor 12; 
20 therefore the compensation algorithm is the same for all image processors. 

FIGURE 2 illustrates an example system 24 showing components used during the 
manufacturing of a digital image sensor. The sensor 12 is coupled to a sensor analysis 
processor 26 during the final stages of sensor production. The system 24 also includes a 
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series of test cards 27-29. In this example, the digital image sensor 12 includes three 
photosites for representing each pixel. The three photosites are red, green, and blue. The 
test cards are a red test card 27, a green test card 28, and a blue test card 29. The sensor 
analysis processor 26 receives the signals generated by each corresponding photosite of 
5 the digital image sensor 12 when exposed to each of the test cards 27-29 and generates a 
profile according to what the digital image sensor 12 should be recording when exposed 
to each of the test cards. The process performed by system 24 is described in more detail 
in FIGURE 3 below. 

The sensor 12 generates an output image signal representing a value from each 
10 photosite that is to be reproduced by correspondingly colored pixel elements on a display 
device that is coupled to the image processor 20. The correspondence between the pixels 
of a sensor and the pixels of a display device is not always an exact 1 : 1 correspondence. 
If it is not a 1:1 correspondence, the image processor 20 adjusts the image recorded by the 
sensor 12 in order to be properly displayed over the display device. 
1 5 FIGURE 3 illustrates the preferred process for generating a profile of the 

operative and inoperative photosites of digital image sensor 12 as performed by the 
system 24 shown in FIGURE 2. First, at block 30, the array of photosites of the digital 
image sensor 12 are exposed one-by-one to a plurality of test cards. The type of tests 
cards used are dependent upon the configuration of the digital image sensor 12. For 
20 example, a black and white digital image sensor only needs two test cards, one black in 
one white. For a digital image sensor that reads color, more specifically red, green, and 
blue, the test cards are one red, one green and one blue. Though these choices are 
preferred, other color combinations would also be used in either case to produce an 
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acceptable profile. Next, at block 32, the digital image sensor 12 generates an output 
signal after exposure to each of the test cards. At block 34, the generated output signal is 
sent to the digital image sensor analysis processor 26. The sensor analysis processor 26, 
at block 36, determines whether there are any malfunctioning components (photosites) 
5 within the digital image sensor 12 by comparing the generated output image signal to 
expected result when exposed to a particular test card. For example, if the test card is a 
red test card, the expected value is a one out of a scale of 0-1, 1 being full on, 0 being off, 
for the red photosites of each pixel. In this example, each pixel of the sensor 12 includes 
three photosites, one that reacts when exposed to green light, a second that registers blue 

10 light and a third that registers red light. Any photosite generating a value different than a 
predetermined threshold from the expected value is determined as malfunctioning. The 
threshold value can vary, but is preferably set closer to the expected value (e.g., 1) than 
the value that would correspond to a malfunction (e.g., 0). 

At block 38, the steps in blocks 30-36 are repeated until all the test cards have 

15 been tested. After all the test cards have been analyzed, at block 40, the sensor analysis 
processor 20 generates a profile of the operative and inoperative photosites of the digital 
image sensor 12. The sensor analysis processor 26 then stores the generated profile in 
non- volatile memory 14 within the digital image sensor 12 for later use in the 
compensation algorithm. 

20 FIGURE 4 illustrates a process performed by the digital image sensor 12 when 

generating images postproduction. First, at block 50, the digital image sensor 12 records 
an image, single image or a video image. Next, at block 52, the recorded image is sent to 
the image processor 20. At block 54, the image sensor profile is retrieved from the image 
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sensor 12 by the image processor 20 (or sent by the sensor 12 to the image processor 20), 
preferably when the image processor 20 is initially connected to the digital image 
sensor 12. Once received, the processor 20 now has a map of all pixels corresponding to 
inoperative photosites that do not respond to a particular color or shade. Next, at 
5 block 56, the image processor 20 adjusts the recorded image according to the image 
sensor profile and the compensation algorithm stored within the processor memory 22. 
An example compensation algorithm is described more detail below with respect to 
FIGURE 5. 

FIGURE 5 shows one embodiment of the compensation algorithm. First, at 
10 block 60, the image processor 20 executing the compensation algorithm, preferably as a 
software program, identifies pixel elements of the recorded image that are to be displayed 
that correspond to malfunctioning photosites of the image sensor according to the image 
sensor profile. Next, at block 60, The processor 20 determines the average color value of 
pixel elements that surround each identified pixel element. At block 64, according to the 
15 algorithm the processor 20 inserts the determined average color value of the surrounding 
pixel elements as the value of the pixel element for the corresponding malfunctioning 
photosite. 

FIGURES 6A-C illustrate an example generation of a sensor's profile after 
exposure to a single test card. FIGURE 6A illustrates a portion 90 of the image signal 
20 generated after exposure to a first test card. FIGURE 6B illustrates an expected 
portion 92 of the image signal that the image sensor 12 should be recording. Upon 
comparing the portion 90 to the expected portion 92 a profile 98 is generated that 
identifies the malfunctioning photosites. For example, the portion 90 includes 
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malfunctioning photosites at locations row 1 column 5, row 2 column 2 and row 3 
column 8. The malfunctioning photosites information is stored as part of the sensor's 
profile. While the profile is depicted in FIGURE 6C as a listing of row and column 
numbers corresponding to malfunctioning photosites, the profile may be stored in other 
ways, such as a table of the type shown in FIGURE 6A. 

FIGURE 7 illustrates a portion 1 10 of a frame of an image generated by the digital 
image sensor 12. In image portion 110, pixel element 112 at row 2 column 2 was 
previously identified as having or producing a malfunctioning result, see FIGURE 6C. 
The image processor 20, according to the compensation algorithm, determines that this 
pixel element 112 is malfunctioning, in accordance with the sensor profile 98. To 
compensate for the error, the method samples the values of pixel elements surrounding 
the malfunctioning pixel element 112, takes the average of those values and inserts that 
average into pixel element 112. 

0.1 + 0.1 + 0.15 + 0.1 + 0.15 + 0.1 + 0.12 + 0.14 = 0.96/8 = 0.12 (1) 
The inserted value is 0.12, see equation (1) above. This is repeated for each base color, 
for example red, green and blue for an RGB color display and then combined to 
present a final adjusted image. 

While the preferred embodiment of the invention has been illustrated and 
described, many changes can be made without departing from the spirit and scope of the 
invention. Accordingly, the scope of the invention is not limited by the disclosure of the 
preferred embodiment. Instead, the invention should be determined entirely by reference 
to the claims that follow. 
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